Destinations v2: Do not dedup raw table #31520

edgao · 2023-10-17T20:39:47Z

Stop deduping the raw table. This necessitates some changes to how T+D works overall. There are also a huge amount of test changes to fix the raw table expectations. (many of these are purely organizational - e.g. combining the dedup/nondedup raw expected records files, since they're now identical)

The final tables remain (almost) identical to previous behavior. The only difference is in how we handle typing failures in the _ab_cdc_deleted_at column. Given that connectors generate that column internally (i.e. we have perfect control over it) I don't think that matters.

previous T+D logic:

insert new raw records into the final table (where loaded_at is null OR deleted_at is not null)
delete duplicate records from final table (where row_number != 1)
delete duplicate records from raw table (where raw_id not in final_table)
CDC deletes (where raw_id not in (select from raw_table where deleted_at is not null))
commit raw table

new logic:

insert new raw records into the final table ((where loaded_at is null OR deleted_at is not null) AND row_number = 1)
1. These are now deduped before they even make it to the final table. This isn't strictly necessary yet, but it will be needed for the merge-based T+D implementation (where each existing final row must match at most one new raw row).
delete duplicate records from final table (where row_number != 1)
CDC deletes (where deleted_at is not null)
1. This is a significant change from the old logic. Because we're keeping all the old raw records, the logic of deleting records with a deletion raw record is no longer valid.
2. But we're still behaving correctly. The old deletion tombstone records are reinserted in step 1 on every T+D invocation, so if the deletion record is the most-recent record then we'll still correctly trigger the delete.
3. This does mean we now only execute CDC deletions if the cast call is successful.
commit raw table

(upcoming merge logic: replace steps 1, 2, and 3 with a single merge statement that does an upsert+CDC deletes in one step)

vercel · 2023-10-17T20:39:53Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 25, 2023 4:03pm

github-actions · 2023-10-17T20:40:10Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

edgao · 2023-10-17T21:14:13Z

...o/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java

-    assertAll(
-        () -> assertEquals("bar", actualRawRecords.get(0).get("_airbyte_data").get("string").asText()),
-        () -> assertEquals("bar", actualFinalRecords.get(0).get(generator.buildColumnId("string").name()).asText()));
+    assertEquals("bar", actualFinalRecords.get(0).get(generator.buildColumnId("string").name()).asText());


there's no longer any reason to assert the raw records, so only assert the final record.

edgao · 2023-10-17T21:14:28Z

...o/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java

        dumpRawTableRecords(streamId),
-        5,
+        6,


this is a behavior change; see https://airbytehq-team.slack.com/archives/C03C4AVJWG4/p1697486280303179

edgao · 2023-10-17T21:15:29Z

...in/java/io/airbyte/integrations/base/destination/typing_deduping/BaseTypingDedupingTest.java

@@ -270,7 +272,7 @@ public void fullRefreshAppend() throws Exception {

    runSync(catalog, messages2);

-    final List<JsonNode> expectedRawRecords2 = readRecords("dat/sync2_expectedrecords_fullrefresh_append_raw.jsonl");
+    final List<JsonNode> expectedRawRecords2 = readRecords("dat/sync2_expectedrecords_append_raw.jsonl");


there's no longer any difference between append and dedup raw records, so merge their expectedrecords files.

edgao · 2023-10-23T21:33:26Z

confirmed via manual test that downgrading from this version to current master results in us re-deduping the raw table (i.e. rollback works as expected). (cc @jbfbell )

airbyte-oss-build-runner · 2023-10-25T16:12:19Z

destination-snowflake test report (commit `b0ca13a992`) - ✅

⏲️ Total pipeline duration: 09mn08s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build destination-snowflake docker image for platform(s) linux/amd64	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

airbyte-oss-build-runner · 2023-10-25T16:19:48Z

destination-bigquery test report (commit `b0ca13a992`) - ✅

⏲️ Total pipeline duration: 07mn16s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build destination-bigquery docker image for platform(s) linux/amd64	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

stuff

5caaf84

octavia-squidington-iii added the area/connectors Connector related issues label Oct 17, 2023

octavia-squidington-iii added the connectors/destination/bigquery label Oct 17, 2023

edgao and others added 3 commits October 17, 2023 20:55

Automated Commit - Formatting Changes

c257f9e

stuff

1bebfd8

more stuff

a9697b8

octavia-squidington-iii added the connectors/destination/snowflake label Oct 17, 2023

more fixes

0ae0e67

edgao force-pushed the edgao/skip_raw_dedup branch from 3a28809 to 0ae0e67 Compare October 17, 2023 22:48

Automated Commit - Formatting Changes

73c54e7

edgao commented Oct 17, 2023

View reviewed changes

edgao marked this pull request as ready for review October 17, 2023 23:26

edgao requested a review from a team as a code owner October 17, 2023 23:26

This comment was marked as outdated.

Sign in to view

only filter on extracted_at in non-dedup mode

5db5b36

edgao mentioned this pull request Oct 17, 2023

Destinations bigquery+snowflake: Dedup using a merge statement; stop deduping raw table #31476

Closed

This comment was marked as outdated.

Sign in to view

edgao mentioned this pull request Oct 18, 2023

Destination bigquery: Use merge statement to do T+D #31529

Merged

Merge branch 'master' into edgao/skip_raw_dedup

55e5506

vercel bot deployed to Preview October 18, 2023 00:54 View deployment

This comment was marked as outdated.

Sign in to view

edgao marked this pull request as draft October 18, 2023 15:49

edgao added 3 commits October 18, 2023 08:49

test cleanup

f37a463

test cleanup

ae2d132

extracted_at filter in dedup mode

34fc49a

Merge branch 'master' into edgao/skip_raw_dedup

55a4470

edgao marked this pull request as ready for review October 18, 2023 21:39

vercel bot deployed to Preview October 18, 2023 21:40 View deployment

This comment was marked as outdated.

Sign in to view

edgao mentioned this pull request Oct 20, 2023

Destination snowflake: T+D via merge #31683

Merged

gisripa mentioned this pull request Oct 24, 2023

Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec #31686

Merged

gisripa approved these changes Oct 24, 2023

View reviewed changes

Merge branch 'master' into edgao/skip_raw_dedup

3b60147

vercel bot deployed to Preview October 24, 2023 17:04 View deployment

This comment was marked as outdated.

Sign in to view

jbfbell approved these changes Oct 24, 2023

View reviewed changes

edgao added 2 commits October 25, 2023 08:59

Merge branch 'master' into edgao/skip_raw_dedup

fa9f214

version bumps

b0ca13a

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Oct 25, 2023

edgao enabled auto-merge (squash) October 25, 2023 16:02

vercel bot deployed to Preview October 25, 2023 16:03 View deployment

edgao merged commit 148dda1 into master Oct 25, 2023
15 of 17 checks passed

edgao deleted the edgao/skip_raw_dedup branch October 25, 2023 16:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Destinations v2: Do not dedup raw table #31520

Destinations v2: Do not dedup raw table #31520

edgao commented Oct 17, 2023 •

edited

Loading

vercel bot commented Oct 17, 2023 •

edited

Loading

github-actions bot commented Oct 17, 2023

edgao Oct 17, 2023

edgao Oct 17, 2023

edgao Oct 17, 2023

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

edgao commented Oct 23, 2023

This comment was marked as outdated.

This comment was marked as outdated.

airbyte-oss-build-runner commented Oct 25, 2023

airbyte-oss-build-runner commented Oct 25, 2023

Destinations v2: Do not dedup raw table #31520

Destinations v2: Do not dedup raw table #31520

Conversation

edgao commented Oct 17, 2023 • edited Loading

vercel bot commented Oct 17, 2023 • edited Loading

github-actions bot commented Oct 17, 2023

Before Merging a Connector Pull Request

edgao Oct 17, 2023

Choose a reason for hiding this comment

edgao Oct 17, 2023

Choose a reason for hiding this comment

edgao Oct 17, 2023

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

edgao commented Oct 23, 2023

This comment was marked as outdated.

This comment was marked as outdated.

airbyte-oss-build-runner commented Oct 25, 2023

destination-snowflake test report (commit b0ca13a992) - ✅

airbyte-oss-build-runner commented Oct 25, 2023

destination-bigquery test report (commit b0ca13a992) - ✅

edgao commented Oct 17, 2023 •

edited

Loading

vercel bot commented Oct 17, 2023 •

edited

Loading

destination-snowflake test report (commit `b0ca13a992`) - ✅

destination-bigquery test report (commit `b0ca13a992`) - ✅